Ranking Images on Semantic Attributes using Human Computation

We investigate to what extent a large group of human workers is able to produce collaboratively a global ranking of images, based on a single semantic attribute. To this end, we have developed CollaboRank, which is a method that formulates and distributes tasks to human workers, and aggregates their personal rankings into a global ranking. Our results show that human workers can achieve a relatively high consensus, depending on the type of the semantic attribute. 1 Semantic Ranking Problem With the Internet being the largest, and fastest growing image database available, responding adequately to a user query remains a constant challenge. Although the precision of the responses has already improved substantially over the past few years, image search may be further improved by ranking the images in a sensible and understandable way. An attractive solution is to rank the images according to their contents, i.e., semantics, on a single attribute. For instance, when searching for an image of an expensive car, ranking images on the semantic attribute “price” would facilitate the search considerably. We define a semantic ranking problem (SRP) as the problem of obtaining a ranking of images, belonging to the same class, based on a single semantic attribute. Jörgensen identified three types of semantic attributes: perceptual, interpretive, and reactive [5]. Perceptual attributes are directly related to a visual stimulus (e.g., color, shape). Interpretive attributes require both interpretation of perceptual cues and a general level of knowledge or inference from that knowledge (e.g., the artist of a painting). Reactive attributes describe a personal response or emotion (e.g., the attractiveness of a face). While ranking images on perceptual attributes is usually trivial for a computer, the latter two types are more challenging, or even impossible. In the domain of Content Based Image Retrieval this is called the semantic gap [7]. We propose that the semantic gap may be bridged by employing a large group of human workers, i.e., human computation. To investigate to what extent human computation can help solving an SRP, we have developed a method called CollaboRank. CollaboRank may be compared to Matchin [3], in the sense that both methods obtain a ranking of images with the help of human workers, or players. Matchin is an online game that shows two images to two players. Both players should click on the image that they think the other player prefers. When both players choose the same image, it is concluded that that image is more beautiful than the other. A single global ranking of “beautifulness” is computed using TrueSkill [4]. CollaboRank differs fromMatchin because each task concerns a well-defined attribute that relates to the semantics of the image, and not the image itself. This allows us to assume a higher level of transitivity in the global ranking, and therefore allows us to distribute tasks containing more than two images. This also influences how CollaboRank computes a global ranking. The remainder of the paper is as follows. Section 2 describes CollaboRank. Experiments and results are presented in Section 3. Section 4 presents our conclusion and gives directions for future research.